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All participants (applicant, applicant's representative, PTO personnel): 



(1) JAVID A. AMINI . (3) . 

(2) William Hunter . (4) . 

Date of Interview: 22 April 2009 . 

Type: a)^ Telephonic Video Conference 

c)D Personal [copy given to: "!)□ applicant 2)D applicant's representative] 

Exhibit shown or demonstration conducted: d)D Yes e)^ No. 

If Yes, brief description: . 

Claim(s) discussed: 7-50, and the new added claims 41-50 on 3/13/08 . 

Identification of prior art discussed: none . 

Agreement with respect to the claims f)^ was reached. g)D was not reached. h)D N/A. 



Substance of Interview including description of the general nature of what was agreed to if an agreement was 
reached, or any other comments: See Continuation Sheet . 

(A fuller description, if necessary, and a copy of the amendments which the examiner agreed would render the claims 
allowable, if available, must be attached. Also, where no copy of the amendments that would render the claims 
allowable is available, a summary thereof must be attached.) 

THE FORMAL WRITTEN REPLY TO THE LAST OFFICE ACTION MUST INCLUDE THE SUBSTANCE OF THE 
INTERVIEW. (See MPEP Section 713.04). If a reply to the last Office action has already been filed, APPLICANT IS 
GIVEN A NON-EXTENDABLE PERIOD OF THE LONGER OF ONE MONTH OR THIRTY DAYS FROM THIS 
INTERVIEW DATE, OR THE MAILING DATE OF THIS INTERVIEW SUMMARY FORM, WHICHEVER IS LATER, TO 
FILE A STATEMENT OF THE SUBSTANCE OF THE INTERVIEW. See Summary of Record of Interview 
requirements on reverse side or on attached sheet. 



/Javid A Amini/ 

Primary Examiner, Art Unit 2628 



PTOL-41 3 (Rev. 04-03) Interview Summary Paper No. 20090424 



Summary of Record of Interview Requirements 



Manual of Patent Examining Procedure (MPEP), Section 713.04, Substance of Interview Must be Made of Record 

A complete written statement as to the substance of any face-to-face, video conference, or telephone interview with regard to an application must be made of record in the 
application whether or not an agreement with the examiner was reached at the interview. 

Title 37 Code of Federal Regulations (CFR) § 1.133 Interviews 
Paragraph (b) 

In every instance where reconsideration is requested in view of an interview with an examiner, a complete written statement of the reasons presented at the interview as 
warranting favorable action must be filed by the applicant. An interview does not remove the necessity for reply to Office action as specified in §§ 1 .1 1 1, 1 .135. (35 U.S.C. 132) 



37 CFR §1 .2 Business to be transacted in writing. 
All business with the Patent or Trademark Office should be transacted in writing. The personal attendance of applicants or their attorneys or agents at the Patent and 
Trademark Office is unnecessary. The action of the Patent and Trademark Office will be based exclusively on the written record in the Office. No attention will be paid to 
any alleged oral promise, stipulation, or understanding in relation to which there is disagreement or doubt. 



The action of the Patent and Trademark Office cannot be based exclusively on the written record in the Office if that record is itself 
incomplete through the failure to record the substance of interviews. 

It is the responsibility of the applicant or the attorney or agent to make the substance of an interview of record in the application file, unless 
the examiner indicates he or she will do so. It is the examiner's responsibility to see that such a record is made and to correct material inaccuracies 
which bear directly on the question of patentability. 

Examiners must complete an Interview Summary Form for each interview held where a matter of substance has been discussed during the 
interview by checking the appropriate boxes and filling in the blanks. Discussions regarding only procedural matters, directed solely to restriction 
requirements for which interview recordation is otherwise provided for in Section 812.01 of the Manual of Patent Examining Procedure, or pointing 
out typographical errors or unreadable script in Office actions or the like, are excluded from the interview recordation procedures below. Where the 
substance of an interview is completely recorded in an Examiners Amendment, no separate Interview Summary Record is required. 

The Interview Summary Form shall be given an appropriate Paper No., placed in the right hand portion of the file, and listed on the 
"Contents" section of the file wrapper. In a personal interview, a duplicate of the Form is given to the applicant (or attorney or agent) at the 
conclusion of the interview. In the case of a telephone or video-conference interview, the copy is mailed to the applicant's correspondence address 
either with or prior to the next official communication. If additional correspondence from the examiner is not likely before an allowance or if other 
circumstances dictate, the Form should be mailed promptly after the interview rather than with the next official communication. 

The Form provides for recordation of the following information: 

- Application Number (Series Code and Serial Number) 

- Name of applicant 

- Name of examiner 

- Date of interview 

- Type of interview (telephonic, video-conference, or personal) 

- Name of participant(s) (applicant, attorney or agent, examiner, other PTO personnel, etc.) 

- An indication whether or not an exhibit was shown or a demonstration conducted 

- An identification of the specific prior art discussed 

- An indication whether an agreement was reached and if so, a description of the general nature of the agreement (may be by 
attachment of a copy of amendments or claims agreed as being allowable). Note: Agreement as to allowability is tentative and does 
not restrict further action by the examiner to the contrary. 

- The signature of the examiner who conducted the interview (if Form is not an attachment to a signed Office action) 

It is desirable that the examiner orally remind the applicant of his or her obligation to record the substance of the interview of each case. It 
should be noted, however, that the Interview Summary Form will not normally be considered a complete and proper recordation of the interview 
unless it includes, or is supplemented by the applicant or the examiner to include, all of the applicable items required below concerning the 
substance of the interview. 

A complete and proper recordation of the substance of any interview should include at least the following applicable items: 

1) A brief description of the nature of any exhibit shown or any demonstration conducted, 

2) an identification of the claims discussed, 

3) an identification of the specific prior art discussed, 

4) an identification of the principal proposed amendments of a substantive nature discussed, unless these are already described on the 
Interview Summary Form completed by the Examiner, 

5) a brief identification of the general thrust of the principal arguments presented to the examiner, 

(The identification of arguments need not be lengthy or elaborate. A verbatim or highly detailed description of the arguments is not 
required. The identification of the arguments is sufficient if the general nature or thrust of the principal arguments made to the 
examiner can be understood in the context of the application file. Of course, the applicant may desire to emphasize and fully 
describe those arguments which he or she feels were or might be persuasive to the examiner.) 

6) a general indication of any other pertinent matters discussed, and 

7) if appropriate, the general results or outcome of the interview unless already described in the Interview Summary Form completed by 
the examiner. 

Examiners are expected to carefully review the applicant's record of the substance of an interview. If the record is not complete and 
accurate, the examiner will give the applicant an extendable one month time period to correct the record. 

Examiner to Check for Accuracy 

If the claims are allowable for other reasons of record, the examiner should send a letter setting forth the examiner's version of the 
statement attributed to him or her. If the record is complete and accurate, the examiner should place the indication, "Interview Record OK" on the 
paper recording the substance of the interview along with the date and the examiner's initials. 



Continuation Sheet (PTOL-413) 



Application No. 10/676,377 



Continuation of Substance of Interview including description of the general nature of what was agreed to if an 
agreement was reached, or any other comments: Authorization for this examiner's amendment was given in a 
telephone interview with William Hunter on 4/22/2009. 
Amendment to the specification is as follows: 

Please replace paragraph [0005], beginning at page 2, line 15, with the following amended paragraph: 

[0005] The present disclosure includes systems and techniques relating to augmented virtual environments, which 
can be implemented using a machine-readable storage device embodying information indicative of instructions for 
causing one or more machines to perform operations as described. According to an aspect, a three dimensional model 
of an environment can be generated from range sensor information representing a height field for the environment. 
Position and orientation information of at least one image sensor in the environment can be tracked with respect to the 
three dimensional model in real-time. Real-time video imagery information from the at least one image sensor can be 
projected onto the three dimensional model based on the tracked position and orientation information, and the three 
dimensional model can be visualized with the projected real-time video imagery. Additionally, generating the three 
dimensional model can involve parametric fitting of geometric primitives to the range sensor information. 

The amendment filed after final on 9/15/2008 has been entered, the amended claims are 45-48, and 50, see below: 
Examiner's notes: the claims amendment filed on 3/13/2008 page 5 disclosed claims 40-49 cancelled, but Applicant 
added new claims 41-50, which the added new claims should have been numbered as 50-59. The renumbering of 
claims is corrected in this action. 

The limitations of dependent claims 41 , 43, and 49 are inserted into the body of their independent claims 29, 37, and 
45, see below: 

25. The method of claim 29, wherein the surface comprises a two dimensional surface. 

29. A method comprising: obtaining a three dimensional model of a three dimensional environment, the three 
dimensional model generated from range sensor information representing a height field for the three dimensional 
environment; 

identifying in real time a region in motion with respect to a background image in real- time video imagery information 
from at least one image sensor having associated position and orientation information with respect to the three 
dimensional model, the background image comprising a single distribution background dynamically modeled from a 
time average of the real-time video imagery information; 

placing a surface that corresponds to the moving region in the three dimensional model, wherein placing the surface 
comprises casting a ray from an optical center, corresponding to the real-time video imagery information, to a bottom 
point of the moving region in an image plane in the three dimensional model, and determining a position, an orientation 
and a size of the surface based on the ray, a ground plane in the three dimensional model, and the moving region; 
projecting the real-time video imagery information onto the three dimensional model, including the surface, based on 
the position and orientation information; and 

visualizing the three dimensional model with the projected real-time video imagery; wherein identifying a region in 
motion in real time comprises subtracting the background image from the real-time video imagery information, 
identifying a foreground object in the subtracted real-time video imagery information, validating the foreground object 
by correlation matching between identified objects in neighboring image frames, and outputting the validated 
foreground object; wherein identifying a foreground object comprises identifying the foreground object in the subtracted 
real-time video imagery information using a histogram-based threshold and a noise filter. 

30. The method of claim 29, further comprising tracking the position and orientation information of the at least one 
image sensor in the environment with respect to the three dimensional model in real-time. 

31. The method of claim 30, wherein obtaining a three dimensional model of a three dimensional environment 
comprises generating the three dimensional model of the three dimensional environment. 

33. The system of claim 37, wherein the surface comprises a two dimensional surface. 

37. An augmented virtual environment system comprising: an object detection and tracking component that identifies in 
real time a region in motion with respect to a background image in real-time video imagery information from at least 
one image sensor having associated position and orientation information with respect to a three dimensional model of 
a three dimensional environment, the three dimensional model generated from range sensor information representing a 
height field for the three dimensional environment, the background image comprising a single distribution background 
dynamically modeled from a time average of the real-time video imagery information, and places a surface that 
corresponds to the moving region with respect to the three dimensional model, wherein the object detection and 
tracking component places the surface by performing operations comprising casting a ray from an optical center, 
corresponding to the real-time video imagery information, to a bottom point of the moving region in an image plane in 
the three dimensional model, and determining a position, an orientation and a size of the surface based on the ray, a 
ground plane in the three dimensional model, and the moving region; 

a dynamic fusion imagery projection component that projects the real-time video imagery information onto the three 

dimensional model, including the surface, based on the position and orientation information; and 

a visualization sub-system that visualizes the three dimensional model with the projected real-time video imagery; 



wherein the object detection and tracking component identifies the moving region by performing operations comprising 
subtracting the background image from the real-time video imagery information, identifying a foreground object in the 
subtracted real-time video imagery 

information, validating the foreground object by correlation matching between identified objects in neighboring image 
frames, and outputting the validated foreground object; wherein identifying a foreground object comprises identifying 
the foreground object in the subtracted real-time video imagery information using a histogram-based threshold and a 
noise filter. 

38. The system of claim 37, further comprising a tracking sensor system that integrates visual input, global navigational 
satellite system receiver input, and inertial orientation sensor input to obtain the position and the orientation information 
associated with the at least one image sensor in real time in conjunction with the real-time video imagery. 

39. The system of claim 38, further comprising a model construction component that generates the three dimensional 
model of the three dimensional environment. 

41. (Canceled) 

42. The method of claim 41 , wherein identifying a region in motion in real time further comprises estimating the 
background image by modeling the background image as a temporal pixel average of five recent image frames in the 
real-time video imagery information. 

43. (Canceled) 

44. The system of claim 43, wherein identifying a region in motion in real time further comprises estimating the 
background image by modeling the background image as a temporal pixel average of five recent image frames in the 
real-time video imagery information. 

45. A machine-readable storage device [medium] embodying information indicative of instructions for causing one or 
more machines to perform operations comprising: 

obtaining a three dimensional model of a three dimensional environment, the three dimensional model generated from 
range sensor information representing a height field for the three dimensional environment; 

identifying in real time a region in motion with respect to a background image in real- time video imagery information 
from at least one image sensor having associated position and orientation information with respect to the three 
dimensional model, the background image comprising a single distribution background dynamically modeled from a 
time average of the real-time video imagery information; 

placing a surface that corresponds to the moving region in the three dimensional model, wherein placing the surface 
comprises casting a ray from an optical center, corresponding to the real-time video imagery information, to a bottom 
point of the moving region in an image plane in the three dimensional model, and determining a position, an orientation 
and a size of the surface based on the ray, a ground plane in the three dimensional model, and the moving region; 
projecting the real-time video imagery information onto the three dimensional model, including the surface, based on 
the position and orientation information; and 

visualizing the three dimensional model with the projected real-time video imagery; wherein identifying a region in 
motion in real time comprises subtracting the background image from the real-time video imagery information, 
identifying a foreground object in the subtracted real-time video imagery information, validating the foreground object 
by correlation matching between identified objects in neighboring image frames, and outputting the validated 
foreground object; wherein identifying a foreground object comprises identifying the foreground object in the subtracted 
real-time video imagery information using a histogram-based threshold and a noise filter. 

46. The machine-readable storage device [medium] of claim 45, wherein the surface comprises a two dimensional 
surface. 

47. The machine-readable storage device [medium] of claim 45, further comprising tracking the position and 
orientation information of the at least one image sensor in the environment with respect to the three dimensional model 
in real-time. 

48. The machine-readable storage device [medium] of claim 47, wherein obtaining a three dimensional model of a 
three dimensional environment comprises generating the three dimensional model of the three dimensional 
environment. 

49. (Cancelled) 

50. The machine-readable storage device [medium] of claim 45, wherein identifying a region in motion in real time 
further comprises estimating the background image by modeling the background image as a temporal pixel average of 
five recent image frames in the real-time video imagery information. . 



